Skip to content

Conversation

@Laikulo
Copy link

@Laikulo Laikulo commented Jun 14, 2025

This adds support for RKNN-based inference of the YOLO2-based model.
RKNN uses its own model format, which (in this case) is made by converting a preexisting ONNX one.

In addition to a new RKNNNet in ml_api, this also includes.

2 scripts for creating RKNN models:

  • convert_to_rknn converts a single model from ONNX to RKNN
  • make_rknn_images converts an ONNX model into a series of RKNN models, targeting different parts.

Two new containers:

  • rknn_toolkit: Debian-based with the full rknn toolkit and a copy of make_rknn_images, meant for batch-converting easily.
  • base_arm64_rknn: An arm64/aarch64 base with the RKNNLite runtime, and rknpu2 library preinstalled.

TODOs:

  • Docstrings in rknn.py
  • Documentation on how to convert models, and build an ml container with a given target model.
  • Docs about running rknnnet in a container (currently needs --privileged for devicetree and possibly some other ioctls)
  • Determine project python version remove "too soon" features if needed.

@Laikulo Laikulo force-pushed the rknn branch 6 times, most recently from 8e5c762 to 31e19e4 Compare June 15, 2025 03:43
@Laikulo
Copy link
Author

Laikulo commented Jun 15, 2025

Vermin indicates that release is 3.8, and so is this branch.

aaron@Skofnung:~/vcs/obico-server> vermin .
Tips:
- Generic or literal annotations might be in use. If so, try using: --eval-annotations
  But check the caveat section: https://github.com/netromdk/vermin#caveats
- You're using potentially backported modules: argparse, dataclasses, enum, importlib, typing
  If so, try using the following for better results: --backport argparse --backport dataclasses --backport enum --backport importlib --backport typing
- Since '# novm' or '# novermin' weren't used, a speedup can be achieved using: --no-parse-comments
(disable using: --no-tips)

Minimum required versions: 3.8
Incompatible versions:     2
aaron@Skofnung:~/vcs/obico-server> git switch rknn
Switched to branch 'rknn'
Your branch is up to date with 'origin/rknn'.
aaron@Skofnung:~/vcs/obico-server> vermin .
Tips:
- Generic or literal annotations might be in use. If so, try using: --eval-annotations
  But check the caveat section: https://github.com/netromdk/vermin#caveats
- You're using potentially backported modules: argparse, dataclasses, enum, importlib, typing
  If so, try using the following for better results: --backport argparse --backport dataclasses --backport enum --backport importlib --backport typing
- Since '# novm' or '# novermin' weren't used, a speedup can be achieved using: --no-parse-comments
(disable using: --no-tips)

Minimum required versions: 3.8
Incompatible versions:     2

@Laikulo Laikulo marked this pull request as ready for review June 15, 2025 03:58
@Laikulo
Copy link
Author

Laikulo commented Jun 15, 2025

Things to discuss:

  • nms_cpu and the post-processing are now used by onnx and rknn, should they be moved into geometry or some other module?
  • The RKNN models are target specific, I'm not super happy about just asking the end user to make sure that one is available at build-time...
  • The ONNX/Darknet weights are delivered by CDN, should the RKNN ones be done so as well?
  • Docker and Podman both require (different!) options to allow the ml_api container to interact with the NPU, which the included docker-composes don't handle. (Do they support inheritance or something like that?)

@Laikulo
Copy link
Author

Laikulo commented Jun 15, 2025

Of note: this may also address #842, since the RK3588 has the same NPU series. I don't have any boards with that part, I can't confirm.

@kennethjiang
Copy link
Contributor

Oh one thing I should have noticed sooner - the PR needs to use "master" branch as the base. We don't directly merge to the "release" branch.

Can you change the base of this PR to "master"? Thank you!

@Laikulo Laikulo changed the base branch from release to master June 15, 2025 14:27
@Laikulo
Copy link
Author

Laikulo commented Jun 15, 2025

Oh one thing I should have noticed sooner - the PR needs to use "master" branch as the base. We don't directly merge to the "release" branch.

Can you change the base of this PR to "master"? Thank you!

Rebased to master, I missed it too.

@Laikulo Laikulo force-pushed the rknn branch 2 times, most recently from fb82959 to fa0d009 Compare June 15, 2025 14:53
@Laikulo
Copy link
Author

Laikulo commented Jun 15, 2025

TODO:

  • Update docs to indicate that the 'docker-compose' in debian/armbian doesn't support the options used to allow RKNN to read firmware info.
  • Add step-by-step for running ml_api or the entire stack on an Radza Zero3 E (the ref board I have been using.

@Laikulo
Copy link
Author

Laikulo commented Jun 15, 2025

TODO:

  • Fix broken checked_call in rknn (looks like args/kwargs not behaving as expected)
    /draft

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants